Information processing apparatus and non-transitory computer readable medium storing information processing program

ABSTRACT

An information processing apparatus includes a character recognition unit that recognizes a handwritten character in creating a tag which is attachable to a mount, a sound recognition unit that recognizes a sound, and a correction unit that corrects a recognition result obtained by the character recognition unit by using a recognition result of the sound recognition unit.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is based on and claims priority under 35 USC 119 fromJapanese Patent Application No. 2017-200107 filed Oct. 16, 2017.

BACKGROUND Technical Field

The present invention relates to an information processing apparatus anda non-transitory computer readable medium storing an informationprocessing program.

SUMMARY

According to an aspect of the invention, there is provided aninformation processing apparatus including a character recognition unitthat recognizes a handwritten character in creating a tag which isattachable to a mount, a sound recognition unit that recognizes a sound,and a correction unit that corrects a recognition result obtained by thecharacter recognition unit by using a recognition result of the soundrecognition unit.

BRIEF DESCRIPTION OF THE DRAWINGS

Exemplary embodiment(s) of the present invention will be described indetail based on the following figures, wherein:

FIG. 1 is a conceptual module configuration diagram illustrating aconfiguration example according to this exemplary embodiment;

FIG. 2 is a diagram illustrating a configuration example of a tag systemusing this exemplary embodiment;

FIGS. 3A to 3C are diagrams respectively illustrating usage examples ofan electronic tag terminal and a shared screen terminal in a conferenceroom or the like used in this exemplary embodiment;

FIG. 4 is a diagram illustrating an example of a data structure of ahandwritten recognition result table;

FIG. 5 is a diagram illustrating an example of a data structure of asound recognition result table;

FIG. 6 is a diagram illustrating an example of a data structure of a taginformation table;

FIG. 7 is a flowchart illustrating a processing example according tothis exemplary embodiment;

FIGS. 8A and 8B are diagrams illustrating a processing example accordingto this exemplary embodiment;

FIG. 9 is a diagram illustrating a processing example according to thisexemplary embodiment;

FIG. 10 is a flowchart illustrating a processing example according tothis exemplary embodiment;

FIG. 11 is a diagram illustrating a processing example according to thisexemplary embodiment;

FIG. 12 is a flowchart illustrating a processing example according tothis exemplary embodiment;

FIG. 13 is a flowchart illustrating a processing example according tothis exemplary embodiment;

FIG. 14 is a flowchart illustrating a processing example according tothis exemplary embodiment; and

FIG. 15 is a block diagram illustrating a hardware configuration exampleof a computer for realizing this exemplary embodiment.

DETAILED DESCRIPTION

Hereinafter, an example of an exemplary embodiment in realizing theinvention will be described with reference to the accompanying drawings.

FIG. 1 is a conceptual module configuration diagram illustrating aconfiguration example of this exemplary embodiment.

Meanwhile, the term “module” refers to components such as software(computer programs) and hardware which are typically capable of beinglogically separated. Consequently, the term “module” in this exemplaryembodiment not only refers to modules in a computer program, but also tomodules in a hardware configuration. Thus, this exemplary embodimentalso serves as a description of a computer program (a program thatcauses a computer to execute respective operations, a program thatcauses a computer to function as respective units, or a program thatcauses a computer to realize respective functions), a system, and amethod for inducing functionality as such modules. Meanwhile, althoughterms like “store” and “record” and their equivalents may be used in thedescription for the sake of convenience, these terms mean that a storageapparatus is made to store information or that control is applied tocause a storage apparatus to store information in the case where theexemplary embodiment is a computer program. In addition, while modulesmay be made to correspond with function on a one-to-one basis, someimplementations may be configured such that one program constitutes onemodule, such that one program constitutes multiple modules, orconversely, such that multiple programs constitute one module. Moreover,plural modules may be executed by one computer, but one module may alsobe executed by plural computers in a distributed or parallel computingenvironment. Meanwhile, a single module may also contain other modules.In addition, the term “connection” may be used hereinafter to denotelogical connections (such as the transmission and reception of data,instructions, a referential relationship between pieces of data, andlog-in) in addition to physical connections. The term “predetermined”refers to something being determined prior to the processing inquestion, and obviously denotes something that is determined before aprocess according to the exemplary embodiment starts, but may alsodenote something that is determined after a process according to theexemplary embodiment has started but before the processing in question,according to conditions or states at that time, or according toconditions or states up to that time. In the case of plural“predetermined values”, the predetermined values may be respectivelydifferent values, or two or more values (this obviously also includesthe case of all values) which are the same. Additionally, statements tothe effect of “B is conducted in the case of A” are used to denote thata determination is made regarding whether or not A holds true, and B isconducted in the case where it is determined that A holds true. However,this excludes cases where the determination of whether or not A holdstrue may be omitted. Additionally, the case of the listing of thingssuch as “A, B, C” is illustrative listing unless otherwise indicated,and includes a case where only one of them is selected (for example,only A).

In addition, the terms “system” and “apparatus” not only encompassconfigurations in which plural computers, hardware, or apparatus areconnected by a communication medium such as a network (includingconnections that support 1-to-1 communication), but also encompassconfigurations realized by a single computer, hardware, or apparatus.The terms “apparatus” and “system” are used interchangeably. Obviously,the term “system” does not include merely artificially arranged socialconstructs (social systems).

Also, every time a process is conducted by each module or every timeplural processes are conducted within a module, information to beprocessed is retrieved from a storage apparatus, and the processingresults are written back to the storage apparatus after the processing.Consequently, description of the retrieval from a storage apparatusbefore processing and the writing back to a storage apparatus afterprocessing may be omitted in some cases. Meanwhile, the storageapparatus herein may include hard disks, random access memory (RAM), anexternal storage medium, storage apparatus accessed via a communicationlink, and registers, and the like inside a central processing unit(CPU).

A tag generation apparatus 100 which is this exemplary embodimentperforms processing regarding a mount and a tag (also referred to as anelectronic tag, an electronic card, and the like) which are used at theassembly, and includes a tag generation start detection module 105, atag information reception module 110, a character recognition module130, a sound recognition module 135, an association module 140, acharacter recognition result correction module 145, a tag generationmodule 150, and a tag operation detection module 155, as illustrated inthe example of FIG. 1. Meanwhile, the meeting may be a meeting in whichplural persons gather and talk with each other, and examples of theassembly include a conference, a workshop, an idea extraction meeting,an examination meeting, a consultation, a discussion, a gathering, ameeting, and the like. This exemplary embodiment relates to anelectronic tag system (also referred to as an electronic white board andthe like) which uses an electronic tag and a mount.

As illustrated in the example of FIG. 2, a shared screen terminal 200and an electronic tag terminal 250 which include the tag generationapparatus 100 are specifically used at an assembly which is performed bya facilitator (generally, one person) and plural participants by using atag. The participant creates a tag in which an idea and the like arewritten, by using the electronic tag terminal 250 as a participantterminal. In addition, the facilitator also creates a tag in which anidea and the like are written, by using the shared screen terminal 200.

In general, as illustrated in the example of FIG. 3, there are pluralelectronic tag terminals 250 including an electronic tag terminal 250A,an electronic tag terminal 250B, and the like (hereinafter,representatively referred to as the electronic tag terminal 250) whichare owned by the respective participants. The shared screen terminal 200receives a tag from the electronic tag terminal 250 and attaches the tagonto a mount (or a background). The facilitator determines or changesthe position of the tag on the mount or arranges the tags (associates afirst tag and a second tag with each other; also referred to as groupingor the like) by using a shared screen which is a display apparatus ofthe shared screen terminal 200, and performs the creation of taginformation, and the like by itself to progress the assembly. Meanwhile,in this exemplary embodiment, the participants include a facilitator.

The tag generation start detection module 105 is connected to the taginformation reception module 110. The tag generation start detectionmodule 105 detects that a tag has been generated, on the shared screenterminal 200 or the electronic tag terminal 250. For example, it may bedetected that a button for generating a tag has been pressed down.

The tag information reception module 110 includes a text data receptionmodule 115, a handwritten information reception module 120, and a soundinformation reception module 125, and is connected to the tag generationstart detection module 105. The tag information reception module 110receives information (data) to be written in a tag. Here, examples ofthe information to be written in a tag include text data and handwrittendata. Sound data is received as auxiliary data for recognizing thehandwritten data or metadata of the tag.

The text data reception module 115 is connected to the tag generationmodule 150. The text data reception module 115 receives text dataserving as contents of a tag. For example, the text data receptionmodule receives text data which is fed by the participant by using akeyboard (a virtual keyboard displayed on a display is included), amouse, or the like.

The handwritten information reception module 120 is connected to thecharacter recognition module 130. The handwritten information receptionmodule 120 receives handwritten data serving as contents of a tag. Forexample, the handwritten information reception module receiveshandwritten data written on a touch panel by using a finger, a pen, orthe like. The handwritten data as mentioned here is data indicating themotion of a finger, a pen, or the like, and is stroke data. The strokedata as mentioned herein includes positional coordinates of a startingpoint of a stroke using a finger, a pen, or the like, positionalcoordinates of a termination point, and information regarding a writtendate and time (may be year, month, day, hour, minute, second, a timeunit smaller than second, or a combination thereof). The stroke data isa target for character recognition by the character recognition module130, and is text data or figure data written in a tag.

Meanwhile, the handwritten information reception module 120 starts toreceive the handwritten data when the tag generation start detectionmodule 105 detects that the generation of a tag is performed.

The sound information reception module 125 is connected to a soundrecognition module 135 and a tag operation detection module 155. Thesound information reception module 125 receives sound data. For example,the sound data may be digital data of a sound received by a microphoneincluded in a pen for handwriting, a microphone within a wearablecomputer worn on a user's clothes, or the like, in addition to a soundreceived by a microphone 255 included in the electronic tag terminal 250and a microphone 205 included in the shared screen terminal 200. Thesound data as mentioned herein may include date and time informationhaving the sound received. The sound data serves a target for soundrecognition by the sound recognition module 135.

Meanwhile, the sound information reception module 125 starts to receivethe sound data when the tag generation start detection module 105detects that the generation of a tag is performed or when the tagoperation detection module 155 detects that an operation with respect toa tag has been performed.

The character recognition module 130 is connected to the handwritteninformation reception module 120 of the tag information reception module110, and the association module 140. The character recognition module130 recognizes a handwritten character in creating a tag which isattachable to a mount. In addition, the degree of certainty of arecognition result thereof may be calculated. The degree of certainty asmentioned herein refers to probability that the recognition result isthe correct answer, and the reliability of the recognition resultincreases as the value of the degree of certainty becomes larger. Thecharacter recognition module 130 may use, for example, a known characterrecognition technique. Specifically, online handwritten characterrecognition may be used.

The character recognition module 130 generates, for example, ahandwritten recognition result table 400 as result of characterrecognition processing. FIG. 4 is a diagram illustrating an example of adata structure of the handwritten recognition result table 400. Thehandwritten recognition result table 400 includes a tag ID column 405, astarting date and time column 410, a termination date and time column415, a handwritten original data column 420, a user ID column 425, arecognition result 1 column 430, a certainty degree 1 column 435, arecognition result 2 column 440, and a certainty degree 2 column 445.The tag ID column 405 stores information (tag Identification (ID)) foruniquely identifying a tag in this exemplary embodiment. The tag asmentioned herein is a tag on which handwriting has been performed. Thestarting date and time column 410 stores a date and time whenhandwriting is started (may be year, month, day, hour, minute, second, atime unit smaller than second, or a combination thereof). Thetermination date and time column 415 stores a date and time whenhandwriting is terminated. The handwritten original data column 420stores handwritten original data (the above-described stroke data). Theuser ID column 425 stores information (user ID) for uniquely identifyinga user performing handwriting in this exemplary embodiment. Therecognition result 1 column 430 stores a recognition result 1 (arecognition result indicating that the degree of certainty is the firstgrade) which is obtained by the character recognition module 130. Thecertainty degree 1 column 435 stores the degree of certainty 1 of therecognition result 1. The recognition result 2 column 440 stores arecognition result 2 (a recognition result indicating that the degree ofcertainty is the second grade) which is obtained by the characterrecognition module 130. The certainty degree 2 column 445 stores thedegree of certainty 2 of the recognition result 2. In addition, arecognition result and the degree of certainty which indicate that thedegree of certainty is equal to or lower than the third grade may bestored.

The sound recognition module 135 is connected to the sound informationreception module 125 of the tag information reception module 110, andthe association module 140. The sound recognition module 135 recognizesa sound. The sound recognition module 135 may use, for example, a knownsound recognition technique.

In addition, the sound recognition module 135 may recognize a soundrelated to a tag.

For example, the sound recognition module 135 may set a sound generatedduring the handwriting of a target character of the characterrecognition module 130 with respect to a tag, as an object to berecognized.

In addition, for example, the sound recognition module 135 may set asound of an operator creating a tag, as an object to be recognized. Thesound to be recognized is a sound which is received by a microphoneincluded in the electronic tag terminal 250 that creates a tag, thevicinity (the above-described wearable computer is included) of theelectronic tag terminal, a pen for handling the electronic tag terminal250, or the like. Meanwhile, in a case where a tag is created on ashared screen, a sound of a facilitator is set to be an object to berecognized, and thus the sound to be recognized in that case is set tobe a sound which is received by a microphone included in the sharedscreen, the vicinity of the shared screen, a pen for handling the sharedscreen, or the like.

In addition, for example, the sound recognition module 135 may set asound generated during the operation of a tag, as an object to berecognized.

The sound recognition module 135 may set a sound generated during theoperation of a tag by a facilitator or a creator of the tag, as anobject to be recognized.

The sound recognition module 135 generates, for example, a soundrecognition result table 500, as results of sound recognitionprocessing. FIG. 5 is a diagram illustrating an example of a datastructure of the sound recognition result table 500. The soundrecognition result table 500 includes a tag ID column 505, a startingdate and time column 510, a termination date and time column 515, asound original data column 520, a user ID column 525, a recognitionresult 1 column 530, a certainty degree 1 column 535, a recognitionresult 2 column 540, and a certainty degree 2 column 545. The tag IDcolumn 505 stores a tag ID. A tag as mentioned herein refers to a tagfor generating a sound generated during the creation or explanation ofthe tag. The starting date and time column 510 stores a date and timewhen the recording of a sound is started. Meanwhile, the recording ofthe sound may be started by the generation of the sound. The terminationdate and time column 515 stores a date and time when the recording of asound is terminated. The sound original data column 520 stores originaldata of a sound. The user ID column 525 stores a user ID of a user whoutters a sound. The recognition result 1 column 530 stores a recognitionresult 1 obtained by the sound recognition module 135. The certaintydegree 1 column 535 stores the degree of certainty 1 of the recognitionresult 1. The recognition result 2 column 540 stores a recognitionresult 2 obtained by the sound recognition module 135. The certaintydegree 2 column 545 stores the degree of certainty 2 of the recognitionresult 2.

The association module 140 is connected to the character recognitionmodule 130, the sound recognition module 135, and the characterrecognition result correction module 145. The association module 140associates a character recognition result obtained by the characterrecognition module 130 and a sound recognition result obtained by thesound recognition module 135 with each other. For example, the characterrecognition result and the sound recognition result may be associatedwith each other in accordance with a date and time when handwritten datais written and a date and time when a sound is received. In a case wherean operation with respect to a tag is performed, a sound received by thesound information reception module 125 and a character recognitionresult in the tag may be associated with each other.

The character recognition result correction module 145 is connected tothe association module 140 and the tag generation module 150. Thecharacter recognition result correction module 145 corrects recognitionresults obtained by the character recognition module 130 by using therecognition results of the sound recognition module 135. A correctionmethod will be described later by using examples of FIGS. 13 and 14.

The tag generation module 150 is connected to the text data receptionmodule 115 of the tag information reception module 110, and thecharacter recognition result correction module 145. The tag generationmodule 150 generates a tag having the text data, received by the textdata reception module 115, as contents, or a tag having the recognitionresults, corrected by the character recognition result correction module145, as contents.

In addition, the tag generation module 150 may add recognition resultsof the sound recognition module 135, as meta information of a tag.

In addition, the tag generation module 150 may add a figure handwrittenon a tag, as meta information. In this case, a handwritten portion whichis irrecognizable by the character recognition module 130 may be set tobe a figure.

The tag generation module 150 generates, for example, a tag informationtable 600, as tag generation results. FIG. 6 is a diagram illustratingan example of a data structure of the tag information table 600. The taginformation table 600 includes a tag ID column 602, a mount ID column604, an attachment position column 606, a size column 608, a creationdate and time column 610, a creator column 612, an attachment date andtime column 614, a color column 616, a frame line shape column 618, aframe line color column 620, a frame line thickness column 622, abelonging group column 624, a content type column 626, a content column628, a handwritten original data column 630, a creation-case soundrecognition result data column 632, a starting date and time column 634,a termination date and time column 636, an explanation-case soundrecognition result data column 638, a starting date and time column 640,and a termination date and time column 642. The tag ID column 602 storesa tag ID. The mount ID column 604 stores information (mount ID) foruniquely identifying a mount in this exemplary embodiment. That is, amount ID indicates a mount to which a tag having the tag ID is attached.The attachment position column 606 stores a position where the tag isattached. That is, the attachment position column stores an attachmentposition on the mount. For example, the attachment position iscoordinates of the mount in an XY coordinate system. The size column 608stores the size of the tag. For example, in a case where a tag to bedisplayed has a rectangular shape, the width and height of the tag arestored. The creation date and time column 610 stores a date and timewhen the tag is created. The creator column 612 stores a creator (userID) of the tag. Alternatively, the creator column may store informationprocessing apparatus (a device ID of the electronic tag terminal 250 orthe shared screen terminal 200) by which the tag is created. Theattachment date and time column 614 may store a date and time when thetag is attached to a mount. The color column 616 stores the displaycolor of the tag. The frame line shape column 618 stores the shape (asolid line, a dotted line, a dashed line, a wavy line, a double line, orthe like) of a frame line in the display of the tag. The frame linecolor column 620 stores the color of the frame line in the display ofthe tag. The frame line thickness column 622 stores the thickness of theframe line in the display of the tag. The belonging group column 624stores information on a group to which the tag belongs. For example,information indicating whether or not the tag belongs to a group may bestored, or a group ID in a case where the tag belongs to a group,another tag ID belonging to the group, and the like may be stored. Thecontent type column 626 stores information indicating the type ofcontents described in the tag, specifically, “text”, “handwrittencharacter recognition result”, “figure”, “image”, or the like. Thecontent column 628 stores contents written in the tag. For example, thetype of contents is text data in a case where the content type column626 is “text” or “handwritten character recognition result”, the type ofcontents is stroke data in a case where the content type column 626 is“figure”, and the type of contents is image data (image data obtained byan electronic camera or the like) in a case where the content typecolumn 626 is “image”. The handwritten original data column 630 storeshandwritten original data in a case where the content type column 626 is“handwritten character recognition result”. In a case where a tag isdisplayed, text data which is a handwritten character recognition resultmay be displayed, or handwritten character itself may be displayed. Thecreation-case sound recognition result data column 632 stores soundrecognition result data in a case where the tag is created. The startingdate and time column 634 stores a date and time when the recording of asound in the creation-case sound recognition result data column 632 isstarted. The termination date and time column 636 stores a date and timewhen the recording of a sound in the creation-case sound recognitionresult data column 632 is terminated. The explanation-case soundrecognition result data column 638 stores sound recognition result datawhen the tag is explained. The starting date and time column 640 storesa date and time when the recording of a sound in the explanation-casesound recognition result data column 638 is started. The terminationdate and time column 642 stores a date and time when the recording of asound in the explanation-case sound recognition result data column 638is terminated.

The tag operation detection module 155 is connected to the soundinformation reception module 125 of the tag information reception module110. The tag operation detection module 155 detects that an operationwith respect to a tag has been performed on the shared screen terminal200 or the electronic tag terminal 250. For example, an operation ofselecting a tag, an operation of displaying a tag in an enlarged manner(an operation for explaining the tag), and the like may be detected.

FIG. 2 is a diagram illustrating a configuration example of a tag systemusing this exemplary embodiment.

The shared screen terminal 200 (a shared screen terminal 200A and ashared screen terminal 200B) includes the tag generation apparatus 100(a tag generation apparatus 100X and a tag generation apparatus 100Y),and is connected to the microphone 205 (a microphone 205A and amicrophone 205B).

The electronic tag terminal 250 (an electronic tag terminal 250A to anelectronic tag terminal 250E) includes the tag generation apparatus 100(a tag generation apparatus 100A to a tag generation apparatus 100E),and is connected to the microphone 255 (a microphone 255A to amicrophone 255E).

The microphone 205 and the microphone 255 may be respectively embeddedin the shared screen terminal 200 and the electronic tag terminal 250.

The shared screen terminal 200 generally includes a large-screen displayapparatus, and is operated by a facilitator. The display apparatus canbe viewed by all participants. Each electronic tag terminal 250 isoperated by a participant at the assembly, and each participantgenerally owns one electronic tag terminal. For example, a tablet-typeterminal or the like is used as the electronic tag terminal 250.

In the example of FIG. 2, the shared screen terminal 200A, theelectronic tag terminal 250A, the electronic tag terminal 250B, and theelectronic tag terminal 250C are installed in a conference room 280A,and assembly is performed. The shared screen terminal 200A, theelectronic tag terminal 250A, the electronic tag terminal 250B, and theelectronic tag terminal 250C are connected to each other through acommunication line 298. The communication line 298 may be a wirelessline, a wired line, or a combination thereof, and may be, for example,the Internet, an intranet, or the like as communication infrastructure.

In addition, a shared screen terminal 200B, an electronic tag terminal250D, and an electronic tag terminal 250E are installed in a conferenceroom 280B, and assembly is performed. The shared screen terminal 200B,the electronic tag terminal 250D, and the electronic tag terminal 250Eare connected to each other through the communication line 298.

A mount and/or tag information storage apparatus 294, devices in theconference room 280A, and devices in the conference room 280B areconnected to each other through a communication line 299. In addition, afunction of the mount and/or tag information storage apparatus 294 maybe realized as a loud service.

The mount and/or tag information storage apparatus 294 storesinformation on a tag. Each shared screen terminal 200 performsprocessing by using the mount and/or tag information storage apparatus294. In this case, the information on the tag which is stored in themount and/or tag information storage apparatus 294 may be shared by theplural shared screen terminals 200. That is, information on a mount anda tag which is managed by the shared screen terminal 200A may be used bythe shared screen terminal 200B. Particularly, in a case where pluralassemblies (for example, assemblies in the conference room 280A and theconference room 280B) relate to one project, a tag corresponding to thesame task may be attached to a mount in another assembly. Accordingly,at one assembly, it is possible to ascertain conditions of the same taskat another assembly.

For example, a participant generates a tag by using the electronic tagterminal 250. In this case, a handwritten character is described in thetag. In the handwriting, a figure is easily drawn, and variations, suchas the change of the size of character, can be made to a sentence, ascompared to a case where a keyboard or the like is used. That is, it iseasy to reflect a creator's intention and to increase the amount ofinformation in one tag.

However, there are restrictions, such as the inability to receive andtransmit information to the next process or the inability to retrieveinformation, in the state of a handwritten character. Consequently, thecharacter recognition of the handwritten character is performed.

Examples of a character recognition technique for the handwrittencharacter include (1) the recognition of the shape of a character and(2) the recognition of stroke data. However, the accuracy of characterrecognition is not perfect, and the degree of accuracy is particularlyreduced in a case where a figure is mixed.

On the other hand, interpretation may be performed for otherparticipants while performing handwriting on a tag. Thereafter, the tagmay be interpreted. Therefore, in this exemplary embodiment, arecognition result of a handwritten character is corrected byrecognizing a sound by the interpretation, explanation, or the like, sothat the accuracy of the recognition result of the handwritten characteris improved.

FIGS. 3A to 3C are diagrams respectively illustrating usage examples ofthe electronic tag terminal 250 and the shared screen terminal 200 in aconference room or the like used in this exemplary embodiment.

As illustrated in the example of FIG. 3A, participants 311 and 312 and afacilitator 321 are gathered in a conference room or the like. Theparticipant 311 uses the electronic tag terminal 250A, and theparticipant 312 uses the electronic tag terminal 250B. In general, oneterminal apparatus (electronic tag terminal 250A or the like) is givento one participant, and the terminal apparatus is a tablet-type terminalwhich is as large as a notebook (for example, A4, B5, 7 inches to 10inches, or the like) like the electronic tag terminal 250 illustrated inthe example of FIG. 3C, and is operated using a finger, a pen, or thelike. Tag information including text data, a handwritten character, afigure, or the like is created by the participant. Meanwhile, theterminal apparatus is not limited to the tablet-type terminal, and maybe a PC (a notebook PC is included) which includes a keyboard, a mouse,and the like.

A shared screen terminal 200 x illustrated in the example of FIG. 3A isa projector, and displays a mount and a tag. In addition, a sharedscreen terminal 200 y is an electronic white board, and detects themotion of the finger, a pen, or the like of the facilitator 321 toreceive an operation such as the attachment of a child mount or a tag toa mount (table), the movement of the child mount or the tag, and theassociation of the child mount or the tag (grouping). For example, theshared screen terminal 200 y is provided with a pen, and thus operationsfor the mount, the child mount, and the tag are received by detectingthe separation of the pen from a predetermined pen holder (lifting-up ofthe pen in order for the facilitator 321 to operate the pen) anddetecting the position of the tip of the pen (the touch of the pen tipon the shared screen terminal 200 y, or the like). For example, a sensor(a sensor which is turned on by the gravity of the pen, or the like) isprovided in the pen holder, and it may be detected which one of pluralpens (a black pen, a red pen, a blue pen, and the like) is used. Inaddition, the entire screen of the shared screen terminal 200 y is atouch sensor, and a touched position of the display screen and pressuremay be detected. In this example, the turn-on and turn-off of the penare controlled by the pen holder, but the pen holder does notnecessarily need to be provided. The turn-on and turn-off of the pen maybe controlled on the pen side. Regarding the change of color, the colormay be changed by preparing a color palette at a portion on the displayscreen and touching a target color by a pen (or something similar to apen) without requiring a pen holder, or a pen may be equipped with afunction (a button, a slider, or the like) for giving an instruction forchanging color.

In addition, the shared screen terminal 200 may be an electronic boardas illustrated in the example of FIG. 3B. The electronic board generallyincludes a large-screen display apparatus (larger than at least thedisplay apparatus of the electronic tag terminal 250), and the displayapparatus is a touch panel and detects a touched position of the displayscreen and pressure. For example, the screen may be a screen having asize of 80 inches, or the like.

FIG. 7 is a flowchart illustrating a processing example according tothis exemplary embodiment.

In step S702, the tag generation start detection module 105 detects thestart of generation of a tag. For example, the tag generation startdetection module detects the pressing-down of a tag generation startingbutton. As illustrated in an example of FIG. 8A, a user performshandwriting by using a pen 850 on a tag writing screen 800. For example,a character and a figure are written by a hand. When the handwriting isperformed, the user utters a sound. The sound may be a sound forexplanation (a case where a tag is created while giving an explanation),or a soliloquy sound. Processes of steps S704 and S706 and processes ofsteps S708 and S710 are performed in parallel.

In step S704, the handwritten information reception module 120 receiveshandwritten information.

In step S706, the character recognition module 130 performs characterrecognition.

In step S708, the sound information reception module 125 receives soundinformation.

In step S710, the sound recognition module 135 performs soundrecognition.

In step S712, the association module 140 associates a characterrecognition result and a sound recognition result with each other. Adescription will be given using an example of FIG. 9. FIG. 9 illustratesa process of generating a tag.

A handwritten character 910 is generated while a user utters a sound950.

Writing start 914 is positioned after tag creation start 912, thehandwritten character 910 is generated, writing termination 916 ispositioned, and a tag is generated at tag creation termination 920. Ahandwritten information acquisition period 918 ranges from the writingstart 914 to the writing termination 916. An association soundinformation acquisition period 952 ranges from the tag creation start912 to the tag creation termination 920. The association soundinformation acquisition period 952 may include at least the handwritteninformation acquisition period 918.

Date and time information may be added to each of a recognition resultof the handwritten character 910 and a recognition result of the sound950 (the starting date and time column 410 and the termination date andtime column 415 of the handwritten recognition result table 400, thestarting date and time column 510 and the termination date and timecolumn 515 of the sound recognition result table 500), and thus therecognition results of the handwritten character 910 and the sound 950are associated with each other so that the pieces of date and timeinformation thereof are consistent with each other. Meanwhile, as the“consistent” as mentioned herein, the association may be performedexactly using the date and time information. However, in general, apoint in time when the handwritten character 910 is written and a pointin time when the sound 950 may deviate from each other, and thus therecognition result of the sound 950 within a period in a predeterminedrange may be associated with a date and time of the recognition resultof the handwritten character 910.

In step S714, the character recognition result correction module 145corrects the character recognition result by using the sound recognitionresult. For example, “0 (numerical number)” and “O (alphabeticcharacter)” as the character recognition result are hardlydistinguishable from each other. That is, the degree of certainty isreduced, or a difference between both the degrees of certainty isreduced. However, it is pronounced “OK” as a sound, and thus the soundof “O (alphabetic character)” is recognized. The character recognitionresult is fixed as “O (alphabetic character)” by using the soundrecognition result. Specifically, correction processing as illustratedin an example of FIG. 13 or FIG. 14 to be described later is performed.

In step S716, the tag generation module 150 applies the soundrecognition result as meta information. Here, the “application of metainformation” specifically refers to the storage of the sound recognitionresult in the creation-case sound recognition result data column 632 ofthe tag information table 600 illustrated in the example of FIG. 6.

In step S718, the tag generation module 150 generates a tag.

A display example of the tag writing screen 800 after this processing isterminated is as illustrated FIG. 8B. That is, the processing result instep S714 is displayed in a character recognition presentation region820.

The character recognition presentation region 820 and a figurepresentation region 825 are displayed on the tag writing screen 800.Here, “

OK” which is the processing result in step S714 is displayed in thecharacter recognition presentation region 820, and a figure which isirrecognizable as a character is displayed in the figure presentationregion 825. Specifically, the figure is stroke data of which the degreeof certainty of character recognition is lower than a predeterminedthreshold value, and is stroke data in a case where correction cannot beperformed even in a case where a sound recognition result is used.Meanwhile, a character recognition result is displayed at the sameposition of handwriting.

FIG. 10 is a flowchart illustrating a processing example according tothis exemplary embodiment.

In step S1002, the tag operation detection module 155 detects anoperation with respect to a tag. For example, a tag selection operation,an enlargement display operation, and the like are detected.

In step S1004, the sound information reception module 125 receives soundinformation.

In step S1006, the sound recognition module 135 performs soundrecognition.

In step S1008, the association module 140 associates a characterrecognition result and a sound recognition result with each other.Meanwhile, the association as mentioned herein means that handwritingdoes not occur, and thus a character recognition result and a soundrecognition result (a sound recognition result in step S1006) in atarget tag are merely associated with each other. However, a characterin a tag is often readout as it is, and thus the order in a characterrecognition result and the order in a sound recognition result areassociated with each other. An example of FIG. 11 shows a process in acase where a tag is explained.

As a tag operation 1110, operation termination 1120 is positioned afteroperation start 1112. Consequently, a period of a target sound 1150 isan association sound information acquisition period 1152 ranging fromthe operation start 1112 to the operation termination 1120.

In step S1010, the character recognition result correction module 145corrects the character recognition result by using the sound recognitionresult. For example, correction processing as illustrated in an exampleof FIG. 14 to be described later is performed.

In step S1012, the tag generation module 150 applies the soundrecognition result as meta information.

In step S1014, the tag generation module 150 corrects a tag.

FIG. 12 is a flowchart illustrating a processing example according tothis exemplary embodiment. The drawing illustrates a control processingexample of a microphone that acquires a sound in the processing(processing using a sound generated during the explanation of a tag)which is illustrated in the example of FIG. 10.

In step S1202, a tag ID of a tag operated by the shared screen terminal200 is extracted.

In step S1204, the microphone 205 of the shared screen terminal 200 isturned on. In other words, this is to cause a facilitator of the sharedscreen terminal 200 to cope with a case where the tag is explained.

In step S1206, a user who creates the tag having the tag ID isextracted.

In step S1208, the microphone 255 of the electronic tag terminal 250used by the user is turned on. In other words, this is to cause thecreator of the tag to cope with a case where the tag is explained.

At an assembly in which the creator of the tag is supposed to performexplanation, the process of step S1204 may be omitted. At an assembly inwhich the facilitator is supposed to perform explanation, the processesof steps S1206 and S1208 may be omitted. At the beginning of theassembly, a rule regarding who will explain a tag may be set, and theprocessing (the process of step S1204 is omitted, or the processes ofsteps S1206 and S1208 are omitted) which is illustrated in the exampleof FIG. 12 may be performed in accordance with the setting.

FIG. 13 is a flowchart illustrating a processing example according tothis exemplary embodiment. The drawing illustrates a processing examplein which a character recognition result is corrected using a soundrecognition result. The processing illustrated in the example of FIG. 7is performed.

In step S1302, it is determined whether “a degree of certainty 1 of arecognition result 1 of a handwritten character is less than a thresholdvalue A”. The processing proceeds to step S1304 in a case where “thedegree of certainty 1 of the recognition result 1 of the handwrittencharacter is less than the threshold value A”, the processing isterminated in other cases (step S1399). Meanwhile, the processing mayproceed to step S1308 in a case where a difference between the degree ofcertainty 1 and a degree of certainty 2 is less than a predeterminedvalue, and the processing may proceed to step S1312 in other cases.

In step S1304, a sound recognition result at a date and timecorresponding to a date and time when handwriting is performed isextracted.

In step S1306, it is determined whether “the degree of certainty 1 ofthe sound recognition result 1 is less than a threshold value B”. Theprocessing proceeds to step S1308 in a case where “the degree ofcertainty 1 of the sound recognition result 1 is less than the thresholdvalue B”, and the processing proceeds to step S1312 in other cases.Meanwhile, the threshold value A and the threshold value B may be thesame value or may be different values.

In step S1308, a user is prompted to perform correction. In other words,this is because there is a high probability that both the characterrecognition result and the sound recognition result are the incorrectanswer. Meanwhile, the character recognition result or the soundrecognition result may be displayed so as to be selectable as acorrection candidate.

In step S1310, the recognition result of the handwritten character iscorrected in accordance with the user's correction operation.

In step S1312, the recognition result of the handwritten character iscorrected to the recognition result of the sound. In other words, thisis because the sound recognition result can be more reliable than thecharacter recognition result.

FIG. 14 is a flowchart illustrating a processing example according tothis exemplary embodiment. The drawing illustrates a processing examplein which a character recognition result is corrected using a soundrecognition result. The processing illustrated in the examples of FIGS.7 and 10 is performed.

In step S1402, it is determined whether “a degree of certainty 1 of arecognition result 1 of a handwritten character is less than a thresholdvalue A”. The processing proceeds to step S1404 in a case where “thedegree of certainty 1 of the recognition result 1 of the handwrittencharacter is less than the threshold value A”, and the processing isterminated in other cases (step S1499).

In step S1404, character recognition results before and after a targethandwritten character are extracted.

In step S1406, it is determined whether or not a sound recognitionresult includes a combination of extracted characters. The processingproceeds to step S1408 in a case where the sound recognition resultincludes a combination of extracted characters, and the processing isterminated in other cases (step S1499).

In step S1408, a recognition result of the handwritten character iscorrected to a recognition result of a sound.

For example, in a case where a recognition result of the targethandwritten character is “X” (the degree of certainty of “X” is lessthan the threshold value A), the previous character is “A”, and thesubsequent character is “B”, it may be retrieved whether a characterstring (“A?B” in a case of written by a regular expression) of “A”, “anyone character”, and “B” is included in the sound recognition result.

In a case where the character string is included in the soundrecognition result, “X” which is the recognition result of thehandwritten character is corrected to a character surrounded by “A” and“B” included in the sound recognition result, in step S1408.

Meanwhile, the previous and subsequent characters have been set to beone character, but may be plural characters. In addition, a character tobe corrected has been set to be one character, but may be pluralcharacters. In addition, the correction may be performed by performinggrammar processing such as morpheme analysis.

Meanwhile, a hardware configuration of a computer executing a program asthis exemplary embodiment is a general computer as illustrated in FIG.15, and specifically, is a personal computer, a computer that may serveas a server, or the like. That is, as a specific example, a CPU 1501 isused as a processing unit (computational unit), and a RAM 1502, a ROM1503, and an HD 1504 are used as storage devices. As the HD 1504, forexample, a hard disk or a Solid State Drive (SSD) may be used. Thecomputer includes the CPU 1501 that executes programs such as the taggeneration start detection module 105, the tag information receptionmodule 110, the character recognition module 130, the sound recognitionmodule 135, the association module 140, the character recognition resultcorrection module 145, the tag generation module 150, and the tagoperation detection module 155, the RAM 1502 that stores the programsand data, the ROM 1503 that stores programs for starting up thecomputer, and the like, the HD 1504 which is an auxiliary storage device(may be a flash memory or the like) which stores the recognition resultobtained by the character recognition module 130, the recognition resultobtained by the sound recognition module 135, the handwrittenrecognition result table 400, the sound recognition result table 500,the tag information table 600, and the like, a reception apparatus 1506that receives data on the basis of a user's operation (including amotion, a sound, an eye gaze, and the like) with respect to a keyboard,a mouse, a touch screen, a microphone, a camera (including an eye gazedetection camera, and the like), and the like, an output apparatus 1505such as a CRT, a liquid crystal display, or a speaker, a communicationline interface 1507, such as a network interface card, for connection toa communication network, and a bus 1508 for transmitting and receivingdata by connecting the above-mentioned components. Plural computers maybe connected to each other through a network.

Regarding the exemplary embodiment based on a computer program among theabove-described exemplary embodiments, a system having this hardwareconfiguration is caused to read a computer program which is software,and the above-described exemplary embodiment is realized by thecooperation of software and hardware resources.

Meanwhile, a hardware configuration illustrated in FIG. 15 shows oneconfiguration example. This exemplary embodiment is not limited to theconfiguration illustrated in FIG. 15, and the information processingapparatus may be configured such that the modules described in thisexemplary embodiment are capable of being executed. For example, somemodules may be constituted by dedicated hardware (for example, anApplication Specific Integrated Circuit (ASIC) or the like), somemodules may be configured to be provided in an external system andconnected to each other through a communication line, or plural systemseach of which is illustrated in FIG. 15 may be connected to each otherthrough a communication line and operated in cooperation with eachother. In addition, the information processing apparatus may beparticularly incorporated into portable information communicationequipment (including a mobile phone, a smart phone, a mobile equipment,a wearable computer, and the like), an information appliance, a robot, acopying machine, a facsimile, a scanner, a printer, a multifunctionmachine (an image processing apparatus including any two or more of ascanner, a printer, a copying machine, and a facsimile), and the like,in addition to a personal computer.

In addition, in comparison processing in the description of theabove-described exemplary embodiment, “equal to or greater than”, “equalto or less than”, “greater than”, and “smaller (less) than” may berespectively set to be “greater than”, “smaller (less) than”, “equal toor greater than”, and “equal to or less than”, as long as inconsistencydoes not occur in the combinations thereof.

Meanwhile, the programs described above may be provided through arecording medium which stores the programs, or may be provided through acommunication unit. In these cases, for example, the programs describedabove may be interpreted as an invention of “a computer-readablerecording medium that stores programs”.

The “computer-readable recording medium that stores programs” refers toa computer-readable recording medium that stores programs and is usedfor the installation and execution of the programs and the distributionof the programs.

Meanwhile, examples of the recording medium include a digital versatiledisk (DVD) having a format of “DVD-R, DVD-RW, DVD-RAM, or the like”which is a standard developed by the DVD forum or having a format of“DVD+R, DVD+RW, or the like” which is a standard developed by the DVD+RWalliance, a compact disk (CD) having a format of CD read only memory(CD-ROM), CD recordable (CD-R), CD rewritable (CD-RW), or the like, aBlu-ray Disc (registered trademark), a magneto-optical disk (MO), aflexible disk (FD), a magnetic tape, a hard disk, a read only memory(ROM), an electrically erasable programmable ROM (EEPROM (registeredtrademark)), a flash memory, a random access memory (RAM), a securedigital (SD) memory card, and the like.

The above-described programs or some of them may be stored anddistributed by recording on the recording medium. In addition, theprograms may be transmitted through communication, for example, by usinga transmission media of, for example, a wired network which is used fora local area network (LAN), a metropolitan area network (MAN), a widearea network (WAN), the Internet, an intranet, an extranet, and thelike, a wireless communication network, or a combination of these. Theprograms may be carried on carrier waves.

Further, the above-described programs may be a portion or all of otherprograms, or may be recorded on a recording medium along with otherprograms. The programs may be recorded on plural recording media bydividing the programs. The programs may be recorded in any format, suchas compression or encryption, as long as it is possible to restore theprograms.

The foregoing description of the exemplary embodiments of the presentinvention has been provided for the purposes of illustration anddescription. It is not intended to be exhaustive or to limit theinvention to the precise forms disclosed. Obviously, many modificationsand variations will be apparent to practitioners skilled in the art. Theembodiments were chosen and described in order to best explain theprinciples of the invention and its practical applications, therebyenabling others skilled in the art to understand the invention forvarious embodiments and with the various modifications as are suited tothe particular use contemplated. It is intended that the scope of theinvention be defined by the following claims and their equivalents.

What is claimed is:
 1. An information processing apparatus comprising: acharacter recognition unit that recognizes a handwritten character increating a tag which is attachable to a mount; a sound recognition unitthat recognizes a sound; and a correction unit that corrects arecognition result obtained by the character recognition unit by using arecognition result of the sound recognition unit, wherein the soundrecognition unit recognizes a sound related to the tag, wherein thesound recognition unit sets a sound generated during the handwriting ofa target character of the character recognition unit with respect to thetag, as an object to be recognized, wherein the sound recognition unitsets a sound of an operator creating the tag, as an object to berecognized.
 2. The information processing apparatus according to claim1, wherein the sound recognition unit sets a sound generated during theoperation of the tag, as an object to be recognized.
 3. The informationprocessing apparatus according to claim 2, wherein the sound recognitionunit sets a sound generated during the operation of the tag by afacilitator or a creator of the tag, as an object to be recognized. 4.The information processing apparatus according to claim 1, furthercomprising: an addition unit that adds the recognition result of thesound recognition unit as meta information of the tag.
 5. Theinformation processing apparatus according to claim 4, wherein theaddition unit adds a figure handwritten on the tag, as the metainformation.
 6. The information processing apparatus according to claim5, wherein the addition unit sets a handwritten portion which isirrecognizable by the character recognition unit, to be the figure.
 7. Anon-transitory computer readable medium storing an informationprocessing program causing a computer to function as: a characterrecognition unit that recognizes a handwritten character in creating atag which is attachable to a mount; a sound recognition unit thatrecognizes a sound; and a correction unit that corrects a recognitionresult obtained by the character recognition unit by using a recognitionresult of the sound recognition unit, wherein the sound recognition unitrecognizes a sound related to the tag, wherein the sound recognitionunit sets a sound generated during the handwriting of a target characterof the character recognition unit with respect to the tag, as an objectto be recognized, wherein the sound recognition unit sets a sound of anoperator creating the tag, as an object to be recognized.
 8. Aninformation processing apparatus comprising: a character recognitionmeans that recognizes a handwritten character in creating a tag which isattachable to a mount; a sound recognition means that recognizes asound; and a correction means that corrects a recognition resultobtained by the character recognition means by using a recognitionresult of the sound recognition means, wherein the sound recognitionmeans recognizes a sound related to the tag, wherein the soundrecognition means sets a sound generated during the handwriting of atarget character of the character recognition means with respect to thetag, as an object to be recognized, wherein the sound recognition meanssets a sound of an operator creating the tag, as an object to berecognized.